Comparative Analysis of AI Planning Systems

Yolanda Gil, USC/ISI). Four general points of agreement emerged. First, the evaluation of planning systems is difficult. There are many different types and levels of evaluation, and it is necessary to be clear about what the goals of any evaluation are. Besides scientific evaluation within the AI community, there can be evaluation by end users of systems, government funding agencies, scientists in other disciplines, industry, and other entities. Each such evaluation is likely to focus on different criteria, and many criteria are subjective and qualitative. It was noted that comparing planners is similar in difficulty to comparing programming languages (in fact, the input specifications to a planner can be viewed as a programming language). Shlomo Zilberstein (University of Massachusetts) presented a number of evaluation measures. Zilberstein called for a new approach to the evaluation of planners. An integrated system that executes or uses the generated plans should be evaluated instead of simply evaluating the plans that can be produced in isolation. For planning systems included in an agent situated in a changing environment, it’s important to evaluate how the planner affects the performance of the agent in its environment. The planner is viewed as a source of information that is used by an execution architecture to select actions. A planner is only as good as the effect it has on the performance of the agent in its operational environment. This holistic view permits comparisons of agents that plan with those that do ■ The Workshop on Comparative Analysis of AI Planning Systems, held during the 1994 national AI conference, was lively and interesting. Both the theoretical and practical sides of the AI planning community were represented. Several papers contributed to the theoretical analysis of planning algorithms, and others showed the first steps toward convergence between such theoretical work and practical work on the system engineering aspects of working planners.